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Abstract 

The computational complexity of a problem arising in the context of sparse optimization is considered, namely, the projection 
onto the set of fc-cosparse vectors w.r.t. some given matrix ft. It is shown that this projection problem is (strongly) NP-hard, 
even in the special cases where the matrix fl contains only ternary or bipolar coefficients. Interestingly, this is in contrast to the 
projection onto the set of fc-sparse vectors, which is trivially solved by keeping only the k largest coefficients. 
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, I. Introduction 



central problem in compressed sensing (CS), see, e.g., [1], [2], [3], is the task of finding a sparsest solution to an 
. underdetermined linear system, i.e., 

min||a;||o s.t. Ax — b, (Po) 



^ for a given matrix A e R™^" with m < n and right hand side vector b G R™, where ||a;||o denotes the £o-quasi-norm, i.e., 

the number of nonzero entries in x. This problem is known to be strongly NP-hard, cf. [MPS] in [4]; the same is true for the 
variant with Ax = b replaced by \\Ax — b\\2 < £ [5], [6]. 

Two related problems arise in signal and image processing, where the unknown signal x to be estimated from a low- 
Q dimensional observation b — Ax cannot directly be modeled as being sparse. In the most standard approach, x is assumed to 
'— ' be built from the superposition of few building blocks or atoms from an overcomplete dictionary D, i.e., x — Dz, where the 
^ representation vector z is sparse. The problem of minimizing ||2;|lo such that ADz = 6 is obviously also NP-hard. 

The alternative cosparse analysis model [7] assumes that Vlx has many zeros, where $7 is an analysis operator. Typical 
ly^ examples include finite difference operators: they are closely connected to total variation minimization and defined as computing 
the difference between adjacent sample values (for a signal) or pixel values (for an image). The cosparse optimization problem 
> of interest reads 

*^ ' min||r2a;||o s.t. Ax = b, (Co) 

m ! 

and was also shown to be NP-hard [7, Section 4.1]. 
CO Due to the hardness of problem (Po), various heuristics have been developed. A popular approach is well-illustrated by the 
\ [ Iterative Hard Thresholding (IHT) algorithm, which iterates a gradient descent step to decrease the eiTor \\Ax — b||2 and a 
J> hard-thresholding step, i.e., the (Euclidean) projection onto the set of fc-sparse vectors. Under restricted isometry properties 
(RIP) on A, the IHT algorithm can be proven to converge to the solution of (Pq) [10], [11]. A desirable RIP often holds with 
high probability when A has i.i.d. sub-Gaussian entries, see, e.g., [9], but is strongly NP-hard to verify in general [8]. 

Adaptations of IHT and related algorithms to the cosparse analysis setting have been proposed [12], [13], based on a 
general scheme for unions of subspaces [14]. A key step is the projection onto the set of fc-cosparse vectors, as an analogous 
replacement for hard-thresholding. 

The main contribution of this note is to show that this projection is in fact strongly NP-hard in general, which contrasts 
with the extremely simple and fast hard-thresholding operation. 



II. Complexity of Cosparse Projection Problems 

This section is mainly devoted to proving the following central result. 

Theorem 1: Consider any p e IN U {cxo}, p > 1, and let q = p if p < oo and q ~ 1 \f p — oo. Given Jl G Q''^" (r > n), 
u) G Q", and a positive integer A; G IN, it is NP-hard in the strong sense to solve the fc-cosparse £p-norm projection problem 

min { llw - z\\l : \\nz\\o < fc }, (fc-CoSPp) 
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even when u G {0, 1}" (with exactly one nonzero entry) and O contains only entries in {—1, 0, +1} or {—1, 1}, respectively. 
Let us first provide some comments on the statement of the theorem. 

Remark 1: Theorem 1 shows the NP-hardness of (fc-CoSPp) in the strong sense (i.e., (/c-CoSPp) is strongly NP-hard). 
Thus, since the objective is always polynomially bounded (never exceeding or ||a;||3o), not even a fully polynomial-time 

approximation scheme (FPTAS) can exist, unless P=NP. An FPTAS is an algorithm that, given an approximation factor e > 0, 
computes a solution that has an objective function value at most (1 + e) times the minimum value, with running time polynomial 
in the encoding length of the instance and 1/e. Moreover, there also cannot exist a pseudo-polynomial (exact) algorithm, i.e., 
one that runs in time polynomial in the numeric value of the input, but not its encoding length, unless P=NP; for details, 
see [4], and also [18]. 

Remark 2: It is perhaps not immediately clear why the "min" in (fc-CoSPp) is attained, since the constraint set is not 
bounded, in general. Nevertheless, the infimum is finite since the objective is lower-bounded by zero. Moreover, since 2: = is 
always feasible, the optimal value is upper-bounded by Since the level set { z : ||r22||o <k, < ||a; - 2;||^ < } 

is compact, the infimum is attained, justifying the use of "min" instead of "inf" in (fc-CoSPp). (See also Remark 6 below.) 

Before stating the proof of Theorem 1, we need some preliminaries. Let MlNULR5'(A,if) be the problem to decide, for a 
given matrix A e Q™^" and a positive integer e M, whether there exists a vector 2; e R" such that z and at most K 
of the m equalities in the system Az = are violated (MinULR stands for "minimum number of unsatisfied linear relations"). 
This homogeneous equality version, in which the trivial all-zero solution is excluded, was proven to be NP-complete [15], and 
even NP-hard to approximate within any constant factor [16]; more results about the problem's approximation complexity can 
be found in [17]. For the proof of Theorem 1, we utilize the following result: 

Theorem 2 (Theorem 1 and Corollary 2 in [15]): MinULR^(A,K) is strongly NP-hard, even when A contains only ternary 
entries, i.e., A e {-1, 0, -M}'"^", or for bipolar A G {-1, -t-1}'"^". 

The cited results from [15] are in fact more general than the statements we use here in that they also hold for non- 
homogeneous systems, where the right hand side vector is part of the input. Moreover, [15] deals with the complementary 
problem of MinULR^, namely the Maximum Feasible Subsystem (MaxFS) problem (w.r.t. homogeneous linear equality 
systems) in which one asks for the largest possible number of simultaneously satisfied equalities from a given linear system. 
It is easy to see that MinULR and MaxFS are equivalent when solved to optimality (see also [17]). 

Note also that the above two results state NP-hardness in the strong sense, although this is not made explicit in the original 
version [15]. The NP-hardness proofs, however, are by reduction from the Exact Cover by 3-Sets (X3C) problem, which is 
well-known to be strongly NP-hard, see, e.g., [4]. 

Proof of Theorem 1: Let p € IN U {00}, p > I, and q = p ii p < 00 and g = 1 if p = 00. Given an instance {A, K) of 
MinULR^" (w.l.o.g. A e Q''^" with r > n), we will reduce it to n instances of (fc-CoSPp). 

For all i = 1, . . . ,n, we define a fc-cosparse projection instance given hy ft = A, lj = Si and k — K (where denotes 
the i-\h unit vector in Q"). Note that each such instance obviously has encoding length polynomially bounded by that of A 
and K. Writing 

:= min{ ||e, - z||« : ||nz||o < k} and := min ^J, 

we observe that since z = is always a feasible point, f-^,^ < 1 for all i, hence fjf ^ < 1. We claim that /^^' < 1 if and only 
if there exists a nonzero vector z that violates at most k = K equations in Az = 0. 

1) If Z^^-* < 1 (i.e., there exists some i G {1, . . . ,n} such that /•'•^^ < 1) then there exists a vector z such that ||r22;||o < k 
and II Cj — z III < 1 (in particular, Zi ^ 0). But this of course means that at most K equalities in the system Az = are 
violated, i.e., MinULRq {A, K) has a positive answer. 

2) Conversely, assume that there exists a nonzero vector z that violates at most K equations in Az = 0, i.e., ||nz||o < k 
and Mi^XJ'LRq [A, K) has a positive answer. We will prove that 

<min||e,-Az|||<l. (1) 

I, A 

In fact, for an arbitrary scalar A e R, z' := Az obeys ||J7z'||o < A: as well. If z contains only one nonzero component Zj, 
A can be chosen sucht that z' = Cj and consequently = f^^^ = 0, which shows the claim. Thus, there remains to 
deal with the case where z contains at least two nonzero components. Consider some i such that Zj ^ 0. Without loss 
of generality, we can also assume that Zi> Q (otherwise we replace z with — z). We consider the case p < 00 first. It 
holds that 

lie, - \z\\i ^ \\\z\\i - |Az,r + ii - Az,r = iAr(iiz||p - |z,r) + |i - Az,r 

for any A. The function |1 — A^il^* is convex (in A), and it is easy to see that we have |1 — Azil^* < 1 — Azi for 
< A < Consequently, 

|Ar(||-^||^-Nir) + |l-Az,r <l + |Ar(||z||^-|z,|^')-Az,, for 0<A<1. (2) 
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Since 2: contains at least two nonzero components, Q! := |zjp > 0. Sincep > 1, itis easy to seethat |A|Pa— Azj < 

for sufficiently small positive A. In conclusion, there exists a solution \z with ||ei — Az||^ < 1 (and thus /^^^ < 1), 
which proves the reverse direction for p < cx). 
Finally, for the case p = 00, note that 

minlle, — < inin||ej — Az||p for all 1 < p < 00, 

whence we can employ the above reasoning to reach the same conclusion. 
To summarize, we could reduce the NP-hard problem MinULR5'(74, iiT) to n instances of the fc-cosparse projection prob- 
lem (fc-CoSPp), which therefore is NP-hard as well. Moreover, by Theorem 2, NP-hardness of (fc-CoSPp) holds in the strong 
sense, even for H i=A) with ternary or bipolar coefficients and, by the construction above, for u: binary (and indeed, with 
only one nonzero entry). ■ 

Remark 3: The reduction in the proof of Theorem 1 does not work for < p < 1. Indeed, consider the vector with all 
entries equal to 1, z = 1. It is not hard to verify that ||ej — Az||*5 = |1 — A|p + (n — 1)|A|p > 1 for all i and A. Thus, 
mini,A||ei — Az||^ > 1 does not imply z = when p e (0, 1]; vice versa, z 7^ does not imply that mint,A||ei — Az||^ < 1. 

Remark 4: It is noteworthy that the claim in the proof of Theorem 1 is in fact true for every real-valued p > 1. However, 
one may encounter irrational (i.e., not finitely representable) numbers when working with p-th powers for arbitrary real, or 
even rational, p. Since retaining finite rational representations is crucial for NP-hardness proofs, we restricted ourselves to 
integer p (or p = 00). 

The following result is an irmnediate consequence of Theorem 1. 

Corollary 1: It is strongly NP-hard to compute a minimizer for the problem (fc-CoSPp), even under the input data restrictions 
specified in Theorem 1. In particular, it is strongly NP-hard to compute the Euclidean projection 

Hn k{<^) '■= arg min | ||u> — zlk : l|fiz||o < k}. (fc-CoSP) 

Proof: Clearly, if a minimizer was known, we would also know the optimal value of (fc-CoSPp). Hence, computing a 
minimizer is at least as hard as solving (fc-CoSPp), and the complexity results of Theorem 1 carry over directly. In particular, 
computing Ilfi,k{^) is also strongly NP-hard, since Theorem 1 applies to the ^2-norm (j>= 2) and the minimizers of \\u3 — z\\2 
(the objective in (fc-CoSP)) are of course the same as those of ||u> — z\^\. ■ 
Let us comment on a few more subtle aspects concerning (fc-CoSPp) and the proof of Theorem 1. 

Remark 5: The above reduction from MinULRq is an example of what is called a Turing reduction; more commonly used 
are Karp reductions, cf. [4], [18]. In the latter, the known NP-hard problem is reduced to a single instance of the problem 
under consideration. For (fc-CoSPp) with p e > 1 (excluding the case p = 00), we could also obtain a Karp reduction by 
constructing the instance 

J7^:=(A^,...,AT)gQ"x™, Lij-^ ■.^{ej,...,el)e{0,ir\ k := nK, 

which is obviously still polynomially related to the input (A, K) of the given MinULR^" instance. Then, defining 

/f) := mmJ||a;-z||P : \\nz\\o<~k}, 

it is easy to see that fj^^ < n{= \\u}\\^) if and only if MinULR|^(A, K) has a positive answer In particular, a solution can 
be seen to be comprised of the solutions to the n separate (A:-CoSPp)-problems considered in the Turing reduction, stacked on 
top of each other. We chose the Turing reduction for our proof of Theorem 1 because it allows us to conclude NP-hardness 
even if u; is a unit vector, i.e., binary with exactly one nonzero entry, and also for p = 00. 

Remark 6: The optimization problem (fc-CoSPp) could be replaced by its corresponding decision version: 

Does there exist some z 6 R" such that Ha; — z||^ < 7 and ||riz||o < fc? (fc-CoSPp-Dec) 

Indeed, the proof of Theorem 1 corresponds to showing hardness of this decision problem for 7 = 1; the Karp reduction 
sketched in the previous remark would use 7 = n. 

Note that (fc-CoSPp-Dec) is in fact contained in NP (at least for p = 1, 2, or 00): the constraint set { z : ||r2z||o < fc } yields 
(exponentially many) affine subspaces of R" defined by n — /e or more homogeneous equahties, and the projection of some 
u} onto it (w.r.t. ||-||p is clearly equivalent to that onto one (unknown) of these affine subspaces. For p = 2, the (EucUdean) 
projection onto such spaces has a known explicit formula which keeps all entries in the solution rational if the input (fl) is 
rational. Similarly, for p = 1 or p = 00, (fc-CoSPp) can be seen as a linear program over the unknown correct affine subspace; 
hence, here, the solution is also rational. 

Thus, for p = 1, 2, or 00, a certificate of a positive answer exists that has encoding length polynomially bounded by that of 
the input. Hence, by Theorem 1 and the above discussion, we have that (fc-CoSPp-Dec) is in fact strongly NP-complete for 
p = 2 01 p = 00. (The case p = 1 is not covered by Theorem 1, and for the remaining values of p it is not immediately clear 
how one could guarantee the existence of a certificate with polynomially bounded encoding length.) 
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III. Conclusions 

Theorem 1 and Corollary 1 show that no polynomial algorithm to compute the projection onto the set of fc-cosparse vectors 
can exist unless P=NP. 

In theoretical algorithmic appUcations of the Euclidean fc-cosparse projection operation (fc-CoSP) in [12], [13], it had so far 
been assumed that the projection problem (fc-CoSP2) can be approximated efficiently. While Remark 1 refutes this assumption 
to a certain degree, it is not clear whether other (general polynomial-time) approximation algorithms exist that may still be 
useful in practice despite exhibiting theoretical running time bounds that depend exponentially on (at least) the approximation 
quaUty 1/e. 

Moreover, as for most NP-hard problems, there are specific instances which are known to be much easier to handle than the 
general case. For instance, when is the identity (or, more generally, a unitary) matrix, hard-tresholding — i.e., zeroing all but 
the k entries with largest absolute values — achieves the projection onto the fc-cosparse set w.rt. any i'p-norm. Other examples 
include the Euclidean case (p = 2) when H is the ID finite difference operator or the ID fused Lasso operator, respectively: 
the projection is then achieved using dynamic programming [13]. 

For the problem MinULR;^ (or its minimization variant, respectively), strong non-approximability results were derived in 
[16], [15], [17]; for instance, it cannot be approximated within any constant unless P=NP. However, these results do not carry 
over to the fc-cosparse projection problem, since the objectives differ: in the optimization version of MinULRq , we wish to 
minimize the number K of violated equalities, while in (fc-CoSPp), the goal is minimizing the distance of z to a given point 
(under the constraint that the number of nonzeros in ftz does not exceed K). Thus, despite the link between the two problems 
exploited in the proof of Theorem 1, (hypothetical) approximation guarantees for this distance unfortunately do not yield any 
(non-)approximability statements for (fc-CoSPp) by means of those for MlNULR^f . 

Thus, it remains a challenge to find (practically) efficient approximation schemes for the fe-cosparse projection prob- 
lem (fc-CoSPp), or to establish further (perhaps negative) results concerning its approximability. Other open questions are 
the complexity of (A;-CoSPp) for < p < 1, or contairmient in NP of (fc-CoSPp-Dec) for p e ]N\ {1, 2}, cf. Remarks 3 and 6. 
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